Skip to content

feat(bots): onboard reviewer-bot to pinned databricks-bot-engine#862

Merged
eric-wang-1990 merged 3 commits into
mainfrom
onboard-bot-engine
Jul 16, 2026
Merged

feat(bots): onboard reviewer-bot to pinned databricks-bot-engine#862
eric-wang-1990 merged 3 commits into
mainfrom
onboard-bot-engine

Conversation

@eric-wang-1990

@eric-wang-1990 eric-wang-1990 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Onboards both bots on databricks-sql-python to the upstream
databricks/databricks-bot-engine, pinned to engine SHA d24ca21 (post-#107
author flow selected from the issue Type) and fully PAT-free (App-auth).
This repo is the first clean consumer of the engine; onboarding friction is
captured as feedback (see the design doc + engine #97).

Design doc: docs/superpowers/specs/2026-07-14-onboard-bot-engine-design.md.

Reviewer bot (was vendored → now engine reusables, App-auth)

  • Replaced the vendored scripts/reviewer_bot + scripts/shared with thin
    callers of the engine's reviewer-bot.reusable.yml / -followup.reusable.yml.
  • engine-auth: app + no engine-pat: the reusable mints an engine-scoped App
    token from the review-bot App creds (engine PR sqlalchemy > 2.0.0 #100).
  • Trigger unchanged: every non-fork PR.

Engineer bot (new — flow from issue Type)

  • engineer-bot.yml (author, label an issue) + engineer-bot-followup.yml
    (reply on an engineer-bot-labeled PR). Own jobs (they build/run the
    connector), each minting an engine-scoped App token for a PAT-free REF install.
  • Author flow chosen from the issue Type (per engine SQLAlchemy Dialect Compliance Tests #107): Bugbug-fix,
    Featureenhancement, any other type / none ⇒ the .bot/config.yaml
    flow: bug-fix default.
  • environment: azure-prod; explicit poetry install so the agent can run
    poetry run python -m pytest tests/unit to self-verify its red→green fix.
  • .bot/config.yaml + engineer prompts, ported from the engine dogfood and
    rewritten for the connector (src/databricks/sql/, mocked tests/unit).

Secrets (all provisioned)

REVIEW_BOT_APP_ID/PRIVATE_KEY, ENGINEER_BOT_APP_ID/PRIVATE_KEY,
DATABRICKS_HOST, DATABRICKS_TOKEN. No BOT_ENGINE_PAT — App-auth
replaces it.

Prerequisite (engine-repo admin) — ✅ done

The review-bot and engineer-bot GitHub Apps are installed on
databricks-bot-engine with contents: read
(required, else the App-auth
engine install fails for both bots).

Verification

  • All workflow YAML + .bot/config.yaml parse; .bot/ tree complete; vendored
    code fully removed with no dangling references.
  • Engine engine-auth: app install path validated end-to-end (engine PR sqlalchemy > 2.0.0 #100).
  • Known follow-up: on the workflow_dispatch (manual) reviewer path the
    explore-tool read root falls back to the default branch — the automatic
    pull_request path (the normal case) is unaffected. Tracked separately.

This pull request and its description were written by Isaac.

Copilot AI review requested due to automatic review settings July 15, 2026 06:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@eric-wang-1990
eric-wang-1990 marked this pull request as draft July 15, 2026 17:53
@eric-wang-1990

Copy link
Copy Markdown
Contributor Author

🚧 Held as draft pending an engine-side fix.

This onboarding is blocked on databricks/databricks-bot-engine#97: the reusable
reviewer workflow can only install the internal engine via a stored
BOT_ENGINE_PAT, and we've chosen to go PAT-free (mint an engine-scoped
GitHub App token instead). Once #97 lands in a merged engine release we'll:

  1. Bump the pinned engine SHA here to the post-fix commit.
  2. Switch the reviewer workflows to engine-auth: app and drop engine-pat.
  3. Restore the pull_request triggers and mark ready for review.

The engineer-bot onboarding (bug-fix flow, PAT-free via its own job) will be
added to this PR in the same change so both bots land together.

Prerequisite for either bot: the review/engineer GitHub App must be installed on
databricks-bot-engine with contents: read.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Medium · 2 Low

Mostly workflow/config plumbing for onboarding the review + engineer bots; no product (src/) code changes. Main concerns: a TEMP e2e workflow the PR itself marks DO-NOT-MERGE is still in the diff (F1), a stray doubly-nested setup-jfrog/setup-jfrog action that nothing references and duplicates the real one (F2), plus a couple of comment/outcome-string mismatches (F3, F4).

Comment thread .github/workflows/engineer-bot-e2e.yml Outdated
Comment thread .github/actions/setup-jfrog/setup-jfrog/action.yml Outdated
Comment thread .github/workflows/engineer-bot.yml
Comment thread .github/workflows/reviewer-bot.yml Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Onboarding PR is almost entirely CI/workflow/config with no connector src/ changes, so risk is scoped to the GitHub Actions. Two issues worth addressing before merge: a self-declared TEMP "DO NOT MERGE" e2e workflow that is still in the diff (medium), and an orphaned double-nested setup-jfrog/setup-jfrog/action.yml duplicate that nothing references (low). Nit: the reviewer-bot.yml header comment still says the token is passed "as engine-pat to install-bot-engine", but the composite input is engine-token — stale wording, worth a one-line fix.

Comment thread .github/workflows/engineer-bot-e2e.yml Outdated
Comment thread .github/actions/setup-jfrog/setup-jfrog/action.yml Outdated
@eric-wang-1990

Copy link
Copy Markdown
Contributor Author

✅ Ready for review — both bots onboarded and verified live

Squashed to 3 signed-off commits (DCO green). Both bots run against the upstream
databricks-bot-engine (pinned b6205fb), PAT-free (engine-scoped App tokens).

Verified end-to-end:

  • Reviewer — posted a real review on this PR as peco-review-bot (inline findings).
  • Engineer (author) — ran 41 turns to a correct structured no_change_needed
    (via a temporary branch-only harness, since issues/dispatch triggers register
    from main; harness removed).

Key design point (first clean consumer): an external repo cannot uses: the
internal engine's actions/reusable workflows cross-repo. So the engine is
installed via a local install-bot-engine composite that pip installs it over
HTTPS with an App token, and connector deps come from the repo's own
setup-poetry (JFrog mirror, egress-safe). Documented in the design doc + engine
issue #104 / doc-PR #105.

Superseded: the vendored reviewer-bot stack (#832/#833/#834) — closed.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Mostly workflow/config plumbing that reads carefully and is well-commented. One medium concern: the engineer-bot issue-comment case matches no_change, while the prompt and the design doc's live-verification note both use the token no_change_needed — a mismatch that would mislabel a real no-change outcome as a failure. Worth confirming the engine's exact outcome-enum string before merge.

Comment thread .github/workflows/engineer-bot.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Solid infra onboarding PR — no security or correctness issues in the token-minting / gating logic, which is carefully done. One medium concern: the issue-comment case in engineer-bot.yml matches no_change but the author actually emits no_change_needed (per this PR's own prompt and design doc), so no-change runs post a misleading "could not complete a fix" comment. Plus one low: the coverage_pr_url output name looks like a copy-paste from a coverage flow and should be confirmed against the engine's bug-fix publish output.

Comment thread .github/workflows/engineer-bot.yml
Comment thread .github/workflows/engineer-bot.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall — this is CI/workflow + .bot/ prompt plumbing with no src/ changes, so the correctness/test axes are limited. Vendored-code removal is clean (grep confirms no dangling scripts.reviewer_bot/scripts.shared references), token handling in install-bot-engine is careful (job-local git extraheader, masking, SHA pin validation), and the security gates are sound. One low-severity gate inconsistency noted inline.

Nits (no inline anchor needed): (1) install-bot-engine/action.yml hardcodes sdk-version: 0.2.102 / cli-version: 2.1.61 defaults that must be manually kept in sync with the engine at the pinned SHA — the comments flag this, but it's a standing drift risk worth a periodic check. (2) The JFrog OIDC block relies on the friendly ACCESS_TOKEN empty-check for error messaging, but under set -euo pipefail a missing ACTIONS_ID_TOKEN_REQUEST_TOKEN would abort at the curl | jq pipe before that check fires; since a

[...truncated to keep verdict scannable]

Comment thread .github/workflows/engineer-bot.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good overall — a clean, well-documented bot-onboarding PR (workflow YAML, local composite actions, prompts, config; no product code touched). Two low-severity consistency notes: the bot-prelude engine-repo input isn't honored by the engine-token mint step, and the initial reviewer workflow lacks the per-PR concurrency group its siblings have. Security-sensitive paths (untrusted issue/PR input via files + env heredocs, digits-only validation, PAT-free git-auth masking, fork gates) are handled carefully.

Comment thread .github/actions/bot-prelude/action.yml
Comment thread .github/workflows/reviewer-bot.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — clean, well-documented onboarding PR. Vendored-bot deletions are confirmed (no dangling references), the four workflows are internally consistent (shared bot-prelude/install-bot-engine composites, PAT-free App-auth, consistent MODEL_ENDPOINT, bot-login prefixes matching .bot/config.yaml), and the untrusted-input handling (digits-only issue/PR validation, heredoc-delimiter collision guard, issue body via file not shell) is careful. One low-confidence note about the workflow_dispatch checkout ref for the reviewer, posted inline.

Comment thread .github/workflows/reviewer-bot.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Onboarding-only PR: four bot workflows + two local composite actions + .bot/ config/prompts, no connector source touched. Auth (engine-scoped App token via git extraheader, PAT-free), input hardening (digits-only validation, untrusted issue body → file, heredoc-delimiter collision check), and the fork/open-PR/label gates all look sound. One low finding: the actually-pinned engine SHA (d24ca217...) matches nowhere in the PR description or design doc (which cite b6205fb / d780b2d), and is duplicated across all four workflows.

Comment thread .github/workflows/reviewer-bot.yml Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — bot/CI onboarding only, no connector source touched; auth flow, fork gates, and untrusted-input handling are careful and well-commented. One low inline note: engineer-bot-followup.yml passes PR_BASE_SHA but (unlike reviewer-bot-followup.yml) never ensures the base commit is fetched. Meta nit for the author, not inline: the PR description states the engine is pinned to SHA b6205fb, but all four workflows pin d24ca2171d191a652a67f4f43995f0959c9a5791 (the design doc's later update confirms b6205fb was an earlier draft) — update the description so reviewers/maintainers see the engine version that actually runs.

Comment thread .github/workflows/engineer-bot-followup.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Solid, carefully-written bot-onboarding infra — vendored code is fully removed (no dangling refs outside the design doc), setup-poetry supplies the interpreter the engine install needs, and security handling (40-char SHA pin validation, masked tokens, git extraheader instead of in-URL creds, argv-prefix bash allowlist, untrusted-input-to-file) is thorough. One low error-handling nit inline. Nits (summary-only): the PR title/description states the engine is pinned to b6205fb, but all four workflows pin d24ca2171d191a652a67f4f43995f0959c9a5791 (the description's later "Update" sections clarify b6205fb was an earlier draft — worth aligning the title). Similarly, the design doc's "Engine pin" section still names d780b2da… while its own later updates use d24ca217…; harmless but stale.

Comment thread .github/actions/install-bot-engine/action.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, carefully-gated onboarding PR — no connector source changes, and the workflow/token/security logic is well-reasoned and thoroughly commented. One medium consistency gap: the .bot/config.yaml templates (name, branch_prefix, pr_title_template, commit_message_template) are all bug-fix-only, but engineer-bot.yml now routes Feature issues to the enhancement flow, which would emit fix:/bugfix-labelled PRs, commits, and branches for enhancements.

Comment thread .bot/config.yaml Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Infrastructure onboarding PR (bot workflows, two local composite actions, engineer-bot config + prompts, design doc). Well-structured and heavily annotated with security reasoning; the token minting, digits-only issue/PR validation, $GITHUB_ENV heredoc-delimiter guard, SHA-pin validation, and fork/open-PR gates all look sound, and no dangling references to the removed vendored bot code remain in the workflows. One low note on the git diff bash-allowlist claim; otherwise looks good.

Comment thread .bot/config.yaml Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Solid, security-conscious bot-onboarding infra PR — no connector source changes, so no unit-test gap. One low finding: the committed design doc / PR description are stale vs the shipped code on the pinned engine SHA (d24ca21 documented vs d9cb1e6 actually pinned in all four workflows) and the Feature→flow mapping (enhancement documented vs task in code). Two acknowledged items I did not file as findings: the workflow_dispatch reviewer path checks out the default branch rather than the PR head (already called out as a tracked follow-up in the PR description), and the engine-ref SHA is hand-duplicated across four workflows with only a comment enforcing lockstep bumps (a nit — a shared var/CI check would remove the drift risk).

Comment thread docs/superpowers/specs/2026-07-14-onboard-bot-engine-design.md Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — workflow/config/docs onboarding only, no src/ changes, and the four bot workflows are internally consistent (same engine SHA across all four, satisfying the documented lockstep requirement). One low: the design doc and PR description cite engine SHAs (d780b2d, d24ca21) that don't match the d9cb1e63… actually pinned in the workflows. Security-sensitive paths (token minting/masking, fork gates, digits-only input validation, heredoc-delimiter collision guard, argv-prefix bash allowlist) are well-reasoned and correctly implemented.

Comment thread docs/superpowers/specs/2026-07-14-onboard-bot-engine-design.md Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Bot-onboarding infra PR (workflows + .bot/ prompts/config); no product code changes. The security gating, token minting, and shell-injection guards are carefully handled. One medium concern: the author flow can be task (for Feature/Task issues) while the engineer prompts are written only for the bug-fix flow, giving the agent contradictory instructions.

Comment thread .github/workflows/engineer-bot.yml Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Workflow/config-only onboarding PR (no src/ changes); gates, token-minting, and JFrog/OIDC install are carefully built and the security rationale is well-documented. One low note: the engine SHA is duplicated across all four workflows and could be centralized in bot-prelude's engine-ref default. Nit: the PR description says the engine is pinned to d24ca21, but the workflows (and the design-doc update) actually pin d9cb1e63… — worth reconciling the description so the stated pin matches what ships.

Comment thread .github/actions/bot-prelude/action.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — infra-only PR (workflows, composites, .bot/ config + prompts) that is carefully written and, per the design doc, already exercised live for both bots. Verified setup-poetry exists with the expected python-version input and that no dangling references to the removed vendored bot code remain under .github. Only one low-severity note: the engine SHA actually pinned in bot-prelude differs from the SHAs quoted in the PR description/design doc — reconcile the docs.

Comment thread .github/actions/bot-prelude/action.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — a clean, carefully-constructed CI onboarding PR (workflows + local composites + .bot/ prompts/config; no executable connector code changed). Gates, untrusted-input handling, and token scoping are thorough; I verified the vendored bot code is fully removed with no dangling references and that setup-poetry / the referenced composites line up. One low-severity credential-cleanup asymmetry noted inline. Note: the PR description still cites engine pin d24ca21, but the actual bot-prelude default (and the design-doc banner) is 3963f4be76be3b4ac0382ca0d7b6bf9d051929e6 — the code is internally consistent, but the description is stale.

Comment thread .github/actions/install-bot-engine/action.yml

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Looks solid overall — this is well-reasoned bot-onboarding infra (workflows + composites + prompts), with no src/ changes so no unit-test coverage is expected. One medium defense-in-depth gap: the git diff HEAD / git log bash-allowlist prefixes still permit an arbitrary-path write via --output=<abs>, which contradicts the file's own carefully-argued sandbox model (see inline finding).

Comment thread .bot/config.yaml Outdated

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review bot failed — see workflow logs.

Reason: Phase 2 gh api failed: could not find pull request diff: HTTP 503: 503 Service Unavailable (https://api.github.com/repos/databricks/databricks-sql-python/pulls/862)

External consumers can't `uses:` the internal engine's actions cross-repo, so
install the engine via a LOCAL composite that pip-installs it over HTTPS with a
short-lived, engine-scoped GitHub App token (no stored PAT), routed through the
internal JFrog mirror. bot-prelude wraps the shared per-workflow setup (mint bot
token + engine-scoped token + Node + install) and centralizes the engine pin in
one place — engine-ref default, pinned to databricks-bot-engine@3569dae (post
Type-flow #107, REPO_ROOT #115, DCO sign-off #117).

Signed-off-by: Eric Wang <e.wang@databricks.com>
Co-authored-by: Isaac
Signed-off-by: eric-wang-1990 <e.wang@databricks.com>
Reviewer + reviewer-followup as own-job workflows that install the engine via the
local bot-prelude composite (PAT-free App-token auth) and run
reviewer_bot.run_review / .followup. Triggers on every non-fork PR; fork gate in
each job `if:`. .bot/prompts/review/system.md is repo-specific ADDITIVE guidance
appended to the engine's base reviewer prompt.

Signed-off-by: Eric Wang <e.wang@databricks.com>
Co-authored-by: Isaac
Signed-off-by: eric-wang-1990 <e.wang@databricks.com>
Engineer author (label an issue) + followup (review-comment replies) as own-job
workflows: setup-poetry for the connector's test deps, engine install via the
local bot-prelude composite, then engineer_bot.run. Author flow selected from the
issue Type (Bug ⇒ bug-fix; other/none ⇒ .bot/config.yaml default, also bug-fix).
REPO_ROOT points the agent's working tree + .bot/ lookup at the checkout.
.bot/config.yaml (bug-fix, poetry pytest allowlist, flow-aware templates omitted
to use engine defaults) + engineer/engineer-followup prompts. Verified live —
reviewer posts findings; engineer author + followup run end to end.

docs/superpowers/specs/: onboarding design + first-consumer findings trail.

Signed-off-by: Eric Wang <e.wang@databricks.com>
Co-authored-by: Isaac
Signed-off-by: eric-wang-1990 <e.wang@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review bot failed — see workflow logs.

Reason: Phase 2 gh api failed: could not find pull request diff: HTTP 503: 503 Service Unavailable (https://api.github.com/repos/databricks/databricks-sql-python/pulls/862)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engineer-bot Maintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants